It’s all in the face: studies on monitoring users’ experience

We propose to observe users’ spontaneous facial expressions as a method to identify adverse-event occurrences at the user interface level. We discuss an experiment designed to investigate the association between incidents during a word processing task and users’ facial expressions monitored using electromyogram sensor devices. The results suggest that an increase of task difficulty is related to an increase in specific facial muscle activity. A second study is presented where an application offers assistance based on the users’ facial expressions. This study evaluates a social agent that reacts to the users’ spontaneous display of emotions, in an attempt to reduce the frustration caused by usability problems. The results indicate that for highly frustrated users the agent often increased their frustration, while those experiencing a moderate level of frustration stated that it somewhat reduced their frustrations.NSF Research Experiences for Undergraduates (REU) - Program (NSF03-599).INI-GraphicsNet Foundation Fellowship Program.The Virtual Humans Project and SAP Research Center,

Towards sociable virtual humans : multimodal recognition of human input and behavior

One of the biggest obstacles for constructing effective sociable virtual humans lies in the failure of machines to recognize the desires, feelings and intentions of the human user. Virtual humans lack the ability to fully understand and decode the communication signals human users emit when communicating with each other. This article describes our research in overcoming this problem by developing senses for the virtual humans which enables them to hear and understand human speech, localize the human user in front of the display system, recognize hand postures and to recognize the emotional state of the human user by classifying facial expression. We report on the methods needed to perform these tasks in real-time and conclude with an outlook on promising research issues of the future